Papers with theoretical framework
Pelican Soup Framework: A Theoretical Framework for Language Model Capabilities (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing theoretical frameworks for large language models (LLMs) do not explain how pretraining leads to in-context learning. |
| Approach: | They propose a theoretical framework that allows LLMs to generalize to unseen instructions and perform in-context learning even when verbalizers are irrelevant to the task. |
| Outcome: | The proposed framework can be used to analyze LLMs' ability to perform in-context learning . it can be applied to linguistic, psychology, and philosophy tasks . |
Rethinking Data Mixing from the Perspective of Large Language Models (2026.acl-short)
Copied to clipboard
Yuanjian Xu, Tianze Sun, Changwei Xu, XinLong Zhao, Jianing Hao, Ran Chen, Yang Liu, Ruijie Xu, Stephen Chen, Guang Zhang
| Challenge: | Existing methods to mix data with LLMs have relied on domain definitions derived from intuition. |
| Approach: | They propose a reweighting framework that restructures data scheduling as a graph-constrained optimization problem. |
| Outcome: | The proposed framework achieves competitive performance on GPT-2 models. |
Sensitivity as a Complexity Measure for Sequence Classification Tasks (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing complexity metrics provide limited practical insight into complexity differences between tasks. |
| Approach: | They propose a theoretical framework for understanding and predicting the complexity of sequence classification tasks using a new extension of the theory of Boolean function sensitivity. |
| Outcome: | The proposed framework predicts the complexity of sequence classification tasks using a new method . it shows that low-sensitivity functions are easier to learn for LSTMs than lexical classifiers . |
A Theory of Unsupervised Speech Recognition (2023.acl-long)
Copied to clipboard
| Challenge: | Unsupervised speech recognition (ASR) is the problem of learning automatic speech recognition systems from unpaired speech-only and text-only corpora. |
| Approach: | They propose a general theoretical framework to study the properties of pasted macro ‘ASRU’/ systems based on random matrix theory and the theory of neural tangent kernels. |
| Outcome: | The proposed framework proves various learnability conditions and sample complexity bounds on synthetic languages with three classes of transition graphs. |
Adaptive Scaling for Sparse Detection in Information Extraction (P18-1)
Copied to clipboard
| Challenge: | Detection problems involving positive instances are often deficient in information extraction tasks . a number of researches have employed neural network models to solve detection problems . |
| Approach: | They propose an algorithm which can handle positive sparsity problem and directly optimize over F-measure . they borrow the idea of marginal utility from economics and propose a theoretical framework for instance importance measuring . |
| Outcome: | The proposed algorithm improves on positive sparsity problem and over F-measure . it leads to more effective and stable training of neural network based detection models. |
Analyzing Transformers in Embedding Space (2023.acl-long)
Copied to clipboard
| Challenge: | Understanding Transformer-based models has attracted significant attention . a zero-pass approach is feasible for some parameters, and for two-layer attention networks . |
| Approach: | They propose a theoretical framework where parameters of a trained Transformer are interpreted by projecting them into the embedding space. |
| Outcome: | The proposed framework shows that pre-trained and fine-tuned models can be interpreted in embedding space. |
Implicit Behavioral Alignment of Language Agents in High-Stakes Crowd Simulations (2025.emnlp-main)
Copied to clipboard
| Challenge: | Language-driven generative agents have enabled large-scale social simulations with transformative uses, from interpersonal training to aiding global policy-making. |
| Approach: | They propose a framework for persona-environment Behavioral Alignment that iteratively refines agent personas and aligns them with real-world expert benchmarks. |
| Outcome: | The proposed framework greatly enhances behavioral realism and reliability in high-stakes social simulations. |